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(57) Abstract: A system and method Ibr adaptively controls sensitivity, on a pixel-by-pixel basis, of a digital imager. An illumi- 
nation intensity level mapping controller deicrmines a number of pixels of image data having illumination intensity levels within a 
first defined range of illumination intensity levels and determines an illumination intensity level mapping function based upon the 
determined number of pixels within the first defined range of illumination intensity levels. An exposure controller determines a 
number of pixels having illumination intensity levels within a second defined range of illumination intensity levels and determines 
an integration time based upon the dciermined number of pixels having illumination intensity levels within the second defined range 
of illumination intensity levels. A tninslcr control function generation circuit determines a composite transfer control function based 
on the determined integration time and determined illumination intensity level mapping function; determines each transition point 
between a plurality of discrete transfer control functions from the determined integration time and the determined illumination in- 
tensity level mapping function; and imposes the determined transfer control function upon a pixel of the digital imager. 
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AD APTIVE SENSITIVITY CONTROL, ON A PIXEL-BY-PIXEL BASIS, FOR A 

DIGITAL IMAGER 



PRIORITY INFORMATION 
This application claims priority from US Provisional Patent Application, Serial 
Number 60/261,023, filed on January 11, 2001; the entire contents of which are hereby 
incorporated by reference. 

FIELD OF THE PRESENT INVENTION 
The present invention is directed to controlling sensitivity, on a pixel-by-pixel basis, 
of a digital imager. More specifically, the present invention is directed to adaptively 
controlling sensitivity, on a pixel-by-pixel basis, of a digital imager to expand a dynamic 
range of the digital imager. 

BACKGROUND OF THE PRESENT INVENTION 
Conventionally, digital imagers, as noted above, are characterized by a linear 
voltage-to-light response, or transfer function; that is, the digital imager output voltage is 
approximately linearly related to light incident on the digital imager. Specifically, the 
output voltage transfer function is linearly proportional to the intensity of the light incident 
upon the digital imager. This linear transfer function can be characterized by a dynanoic 
range. The dynamic range is defined as the difference between the highest detectable 
illumination intensity of a scene to the lowest detectable illumination intensity of the scene. 
It is well understood that the dynamic range of the transfer function sets the overall dynamic 
range of the digital imager. If ttie dynamic range of a scene exceeds the dynamic range of a 
digital imager, portions of the scene will saturate the digital imager and appear either 
completely black or completely white. This can be problematic for imaging large dynamic 
range scenes, such as outdoor scenes. 

A conventional digital imager includes a plurality of photosensitive elements 
commonly referred to as pixels. The physical realization of flie pixels is eiflier a plurality of 
phototransistors or a plurality of photodiodes functioning as light-detecting elements. In 
operation, the pixel is first reset with a reset voltage that places an electronic charge across 
the capacitance associated with the diode. Electronic charge produced by, for example, a 
photodiode, when exposed to illumination, causes charge of the diode capacitance to 
dissipate in proportion to the incident illumination intensity of the scene. At the end of an 
exposm-e period, the change in diode capacitance charge is detected and the photodiode is 
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reset. The amount of light detected by the photodiode is computed as the difference 

between the reset voltage and the voltage corresponding to the final diode capacitance 

charge. 

The sensitivity of a digital imager, such as that provided in a digital still or video 
5 camera, can in general be defined by the maximum and minimimGi light intensities of a scene 
that can be imaged by the digital imager. The light or illumination intensity of a scene 
captured by a pixel of a digital imager is a value related to an output voltage or amount of 
charge measured at the pixel, photosensitive element. 

There are various situations that can limit the sensitivity of a digital imager. One 

10 example is pixel saturation. Pixel saturation is realized when a photosensitive element, 
pixel, becomes depleted of charge such that no amount of additional incident hght will 
change the output voltage, thus pixel saturation acts to limit the maximum light intensity of 
the scene that can be discretely imaged. In other words, if the saturation of a pixel is an 
illumination intensity level of 100, all illumination intensity levels of the scene that are 

15 above an illumination intensity level of 100 are represented as an illiunination intensity 
level of 100 in the imaged scene, and thus, these illumination intensity levels of the scene 
that are above an illumination intensity level of 100 are not discretely imaged or are not 
discernible because notwithstanding the actual illumination intensity level, above 100, the 
imaged illumination intensity level is 100. 

20 Another example i$ pixel noise. Examples of sources of pixel noise are quantization 

in the conversion of the analog voltage to a digital code, integrated thermal charge 
production, and voltage reset variation when a pixel is not sampled using a true correlated 
double sample. Pixel noise sets the minimmn detectable illumination intensity. Since the 
human eye has a logarithmic response to intensity, in digital imaging, it is usually noise in 

25 the lower intensities that is of concern. 

Both pixel saturation and pixel noise are inherent properties of a digital imager. So, 
these properties limit the physical sensitivity of a digital imager such that a scene, having a 
wide range of light (illumination) intensities, cannot be imaged properly; the highCT light 
(illumination) intensities of the scene are washed out in the imaged scene due to pixel 

30 saturation, or the low light (illumination) intensities of the scene become less resolvable due 
to pixel noise. 

Conventionally, methods have been proposed to nullify these physical limitations of 
a digital imager, such as the manipulation or adjustment of the charge integration function 
of the pixel, also known as the transfer control function of the digital imager. Charge 
35 integration function manipulation or transfer control function manipulation has. 
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conventionally, been realized through the changing of an integration time, Tim, for the 
digital imager. Changing the integration time, Tint, changes the start time of the transfer 
control function or charge integration period, hi other words, changing the integration time 
is a fomi of electronic-irising or exposure control, analogous to controlling the speed of a 
5 shutter on an analog camera. An example of conventional integration time manipulation is 
illustrated in Figures 1 and 2. 

Figure 1 shoves a linear transfer control function wherein an integration time. Tint, is 
near a maximum mtegration time. Integration time, Ti„t, is the time that a control signal 10 
is not set at a reset level. When the control signal 10 is not at a reset value during a frame 

10 period, F, the digital imager causes charge to be transferred or collected from a pixel. As 
shown in Figure 1, the control signal 10 is initially at a reset level (at the beginning of the 
frame period, F); however, the control signal 10 changes to another level, in this case a 
collect level, at a point in time, within the frame period, that is equal to a difference between 
the frame period, F, and the integration time. Tint, (hi Figure 1, the difference is very small.) 

15 The control signal 10 changes back to the reset level at the end of the frame period, F. 

On the other hand, Figure 2 shows a linear transfer control function wherein the 
integration time, Ti„t, is decreased. As noted above, integration time. Tint, is the time that a 
control signal 10 is not set at a reset level. When the control signal 10 is not at a reset value 
during a frame period, F, the digital imager causes charge to be transferred or collected from 

20 a pixel. As shown in Figure 2, the control signal 10 is initially at a reset level (at the 
beginning of the frame period, F); however, the control signal 10 changes to another level, 
in this case a collect level, at a point in time, withm the frame period, that is equal to a 
difference between the frame period, F, and the integration time. Tint. (In Figure 2, the 
difference is approximately equal to half the frame period, F.) The control signal 10 

25 changes back to the reset level at the end of the frame period, F. 

In other words. Figure 2 shows a situation where the control signal 10 is set at a 
level that causes charge to be transferred or collected from the pixel for a shorter duration of 
time than shown in Figure 1. By decreasing the integration time, the low illumination 
intensity levels of the scene being imaged are de-emphasized because not enou^ time is 

30 given to for the charge associated with the low illumination intensity levels of the scene to 
accumulate because the low illumination intensity level does produce charge as quickly as 
the high illmnination intensity levels. 

Another way of looking at the relationship of the integration time, Ti„t, of the digital 
imager to the illumination intensity level of the imaged scene is shown in Figure 3. In 

35 Figure 3, curve A represents a first mapping of pixel output voltage versus illumination 
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intensity level of the imaged scene using a first integration time, Tlmt- Curve B shows a 

second mapping of pixel output voltage versus illumination intensity level of the imaged 

scene using a second iutegration time, T2int. In Figure 3, the second integi-ation time, T2i„t, 

is smaller than the first integration time, Tlim. Thus, as illustrated in Figure 3, as the 

integration time, Tint, is increased, the slope of the map of pixel output voltage versus 

illumination intensity level of the imaged scene is increased, thereby emphasizing the lower 

illununation intensity levels of ftie imaged scene. 

Another conventional way of manipulating the transfer control function is to use a 
stepped or piecewise discrete-time transfer control function. By using a stepped or 
piecewise discrete transfer control function, the mapping of the pixel output voltage versus 
illumination intensity level of the imaged scene can be modified to enable a wider range of 
possible illumination intensity levels of the imaged scene before saturation, while 
emphasizing low illumination intensity levels of the imaged scene. 

An example of a conventional stepped or piecewise discrete-time transfer control 
function is illustrated in Figure 4. In Figure 4, the integration time, Tjnt, is the same value as 
illustrated in Figure 1. However, unlike Figure 1, the level of the control signal 10, in 
Figure 4, is stepped down in a discrete fashion firom a reset level to a collect value. The 
control signal 10 acts as a barrier to charge production or accumulation. At the reset level, a 
complete barrier is realized wherein no charge is produced or accumulated, notwithstanding 
the illumination intensity level of the light incident upon the pixel. In other words, the pixel 
is effectively turned off As the control signal 10, is stepped down fi-om a reset level, the 
effective barrier is lower, thus allowing a rate of production or accumulation of charge to 
effectively realize a gradual increase. 

Specifically, as illustrated in Figure 4, over the course of an exposure period, the 
control signal 10 is applied to the pixel or photodiode capacitance to control charge 
dissipation from the capacitance. The control signal 10 is typically decreased firom the 
starting pixel reset voltage value to, e.g., electrical ground or a collect level, with each 
control signal 10 value at a given time during the exposure period setting the maximum 
charge dissipation of the photodiode. This control signal 10 decrease acts to increase the 
photodiode charge dissipation capabiUty, whereby the pixel can accommodate a higher 
illumination intensity before saturating, and the dynamic range of the pixel is thusly 
increased. This charge dissipation control produces a nonlinear pixel output voltage to 
imaged scene illumination intensity level mapping and a correspondingly expanded 
dynan:uc range of the pixel and the imager. This is seen in Figure 5. 
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Using a step or piecewise discrete- time transfer control function, as illustrated 
in Figure 4, a mapping of the pixel output voltage versus illumination intensity level of the 
imaged scene having a voider range, as illustrated in Figure 5, is produced. In Figure 5, 
curve A shows a mapping of the pixel output voltage versus illimiination intensity level of 
the imaged scene when a linear control transfer function, as illustrated in Figure 1, is used, 
whereas curve B shows a mapping of the pixel output voltage versus illumination intensity 
level of the imaged scene when a stepped or piecewise discrete-time transfer control 
function, as illustrated in Figure 4, is used. As illustrated in Figure 5, curve B has a wider 
range of illumination intensity levels of the imaged scene than curve A. 

Although the physical limitations of digital imagers have been compensated for 
through integration time adjustment and transfer control function (resulting in the 
production of the control signal 10) manipulation, these conventional approaches still fail to 
compensate for the digital imager's physical limitations in an optimal way. 

For example, the conventional processes fail to realize an illumination intensity level 
of the imaged scene range in which no saturated pixels exist. More specifically, as 
illustrated in Figure 6, a histogram of actual illunaination intensity levels of a scene to be 
imaged may include illumination intensity levels from 20 to 200 when the image average is 
50; however, due to pixel saturation, illumination intensity levels of the scene above the 100 
illumination intensity level will not be represented by the digital imager, as illustrated 
Figure 7, because these higher illxmiiriation intensity levels of the scene are all bucketed at 
the 100 illumination intensity level by the digital imager due to pixel saturation, thereby 
washing out the higher illumination intensity levels of the imaged scene. 

Using the conventional processes described above to compensate for the saturated 
pixels by reducing the integration time, a histogram of the illumination intensity levels of 
the imaged scene is realized as illustrated in Figure 8. In Figure 8, the histogram of the 
imaged scene is scaled by a factor of 0.5 such that the image illumination intensity average 
changes from 50 to 25. Using this conventional approach, a greater portion of the higlier 
illxmiination intensity levels of the scene can be represented; however, the image is 
. unacceptably dark, the quantization noise of the entire image has been cut in half, and the 
signal may also have been reduced by a factor of 2 while other noise sources stay fixed in 
magnitude (such as reset noise). 



SUMMARY OF THE PRESENT INVENTION 
A first aspect of the present invention is a method of adaptively controlling 
sensitivity, on a pixel-by-pixel basis, of a digital imager. The method determines a number 
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of pixels of image data having illumination intensity levels within a first defined range of 

illxmiination intensity levels; determines an illmnination intensity level mapping function 

based upon the determined number of pixels within the first defined range of illumination 

intensity levels; determines a transfer control function based on the determined illumination 

intensity level mapping function; and imposes the determined transfer control function upon 

a pixel of the digital imager. 

A second aspect of the present invention is a method of adaptively controlling 
sensitivity, on a pixel-by-pixel basis, of a digital imager. The method detemiines a plurality 
of numbers of pixels, each determined number of pixels being a number of pixels within an 
associated defined range of illumination intensity levels; determines a plurality of 
illumination intensity level mapping functions, each determined illumination intensity level 
mapping function corresponding to one defined range of illumination intensity levels, each 
illumination intensity level mapping function being determined based upon the determined 
number of pixels within the corresponding defined range of illumination intensity levels; 
detemiines a transfer control function based on the plurality of determined illumination 
intensity level mapping functions; and imposes the detennined transfer control function 
upon a pixel of the digital imager. 

A third aspect of the present invention is a method of adaptively controlling 
sensitivity, on a pixel-by-pixel basis, of a dijgital imager. The method determines a number 
of saturated pixels; selects a first illumination intensity level mapping function when the 
determined number of saturated pixels is above a first threshold; determines an number of 
pixels having illumination intensity levels within a defined range of values; selects a second 
illumination intensity level mapping function when the detennined number of pixels is 
below a second threshold; determines a transfer control function based on the selected 
illumination intensity level mapping function; and imposes the detennined transfer control 
function upon a pixel of the digital imager. 

A fourth aspect of the present invention is a method of adaptively controlling 
sensitivity, on a pixel-by-pixel basis, of a digital imager. The method determines a number 
of pixels of image data having illuminatiou intensity levels within a first defined range of 
illumination intensity levels, the first defined range of illumination intensity levels including 
an illumination intensity level corresponding to a pixel saturation value; determines an 
illmnination intensity level mapping fimction based upon the determined nmnber of pixels 
within the first defined range of illxunination intensity levels; determines a number of pixels 
having illumination intensity levels within a second defined range of illumination intensity 
levels, the second defined range of illumination intensity levels including an illumination 
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intensity level corresponding to a minimum illumination intensity level; deteraiines an 
integration time based upon the determined number of pixels having illumination intensity 
levels withiii the second defined range of illximination intensity levels; determines a transfer 
control function based on the determined illumination intensity level mapping function and 
5 the determined integration time; and imposes the determined transfer control function upon 
a pixel of the digital imager. 

A fifth aspect of the present invention is a method of adaptively controlling 
sensitivity, on a pixel-by-pixel basis, of a digital imager. The method selects a first 
illumination intensity level mapping function; determines a first transfer control function 

10 based on the selected first compression; imposes the determined first transfer control 
function upon a pixel of the digital imager; determines a histogram of illumination intensity 
levels of pixels of image data being generated by the digital imager having the determined 
first transfer control function imposed thereon; determines an illxmiination intensity level 
maximum, the illumination intensity level maximum representing a greatest illumination 

15 intensity level for a pixel in a sample forming the histogram; determines a second 
illxmiination intensity level mapping function, based on the determined intensity level 
maximum, the second illimiination intensity level mapping function preventing the 
generation of any saturated pixels and providing a dynamic range of image data enabling 
each level in the histogram to be realized by the digital imager; determines a second transfer 

20 control function based on the determined second illumination intensity level mapping 
function; and imposes the determined second transfer control function upon a pixel of the 
digital imager. 

A sixth aspect of the present invention is a method for determining transition points 
between a plurality of discrete transfer control functions forming a composite transfer 

25 control function. The method determines an integration time; determines an illumination 
intensity level mapping function; determines a composite transfer control function based on 
the determined integration time and determined illumination intensity level mapping 
function; and determines a timing of each transition point between a plurality of discrete 
transfer control functions firom the determined integration time and the determined 

30 illumination intensity level mapping function. 

A seventh aspect of the present invention is a system for adaptively controlling 
sensitivity, on a pixel-by-pixel basis, of a digital imager. The system includes an 
illumination intensity level mapping controller to determine a number of pixels of image 
data having illumination intensity levels within a first defined range of illumination 

35 intensity levels and to determine an illimiination intensity level mapping function based 
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uponthedeterminednumber of pixels within the first defined range of illumination 

intensity levels and a transfer control function generation circuit to determine a transfer 

control function based on the determined illumination intensity level mapping function and 

to impose the determined transfer control function upon a pixel of the digital imager. 

5 An eighth aspect of the present invention is a system for adaptively controlling 

sensitivity, on a pixel-by-pixel basis, of a digital imager. The system includes an 

illumination intensity level mapping controller to determine a plurality of number of pixels, 

each determined number of pixels corresponding to one defined range of illumination 

intensity levels, and to determine a plurality of illumination intensity level mapping 

10 functions, each determined illumination intensity level mapping function corresponding to 
one defmed range of illumination intensity levels, each illumination intensity level mapping 
function being determined based upon the determined niunber of pixels within the 
corresponding defined range of illmnination intensity levels and a transfer control function 
generation circuit to determine a transfer control function based on the plurality of 

15 determined illumination intensity level mapping functions and to impose the determined 
transfer control function upon a pixel of the digital imager. 

A ninth aspect of the present invention is a system for adaptively controlling 
sensitivity, on a pixel-by-pixel basis, of a digital imager. The system includes an 
illumination intensity level mapping controller to determine a nmnber of saturated pixels 

20 and to select a first illumination intensity level mapping function when the determined 
number of saturated pixels is above a first threshold. The illumination intensity level 
mapping controller determines a number of pixels having illumination intensity levels 
within a defined range of values and selects a second illumination intensity level mapping 
function when the determined number of pixels is below a second threshold. The system 

25 further includes a transfer control function generation circuit to determine a transfer control 
function based on the selected illumination intensity level mapping function and to impose 
the determined transfer control function upon a pixel of the digital imager. 

A tenth aspect of the present invention is a system for adaptively controlling 
sensitivity, on a pixel-by-pixel basis, of a digital imager. The system includes an 

30 illumination intensity level mapping controller to determine a number of pixels of image 
data having illumination intensity levels within a first defined range of illumination 
intensity levels, the first defined range of illumination intensity levels including an 
illumination intensity level corresponding to a pixel saturation value, and to determine an 
illumination intensity level mapping fimction based upon the determined number of pixels 

35 within the first defined range of illiunination intensity levels; an exposure controller to 
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determine a number of pixels having illvimination intensity levels within a second 
defined range of illumination intensity levels, the second defined range of illumination 
intensity levels including an illumination intensity level corresponding to a minimum 
illumination intensity level, and to determine an integration time based upon the determined 
5 number of pixels having illimiination intensity levels w^ithin the second defined range of 
illumination intensity levels; and a transfer control fimction generation circuit to determine 
a transfer control function based on the determined illumination intensity level mapping 
function and the determined integration time and to impose the determined transfer control 
fimction upon a pixel of the digital imager. 

10 An eleventh aspect of the present invention is a system for adaptively controlling 

sensitivity, on a pixel-by-pixel basis, of a digital imager. The system includes an 
illumination intensity level mapping controller to select a first illumination intensity level 
mapping function and a transfer control function generation circuit to determine a first 
transfer control function based on the selected first compression and to impose the 

15 determined first transfer control function upon a pixel of the digital imager. The 
illimiination intensity level mapping controller determines a histogram of illumination 
intensity levels of pixels of image data being generated by the digital imager having the 
determined first transfer control function imposed thereon and determines an illumination 
intensity level maximxmi, the illumination intensity level maximum representing a greatest 

20 illumination intensity level for a pixel in a sample forming the histogram. The illumination 
intensity level mapping controller determines a second illumination intensity level mapping 
function, based on the determined intensity level maximum, the second illumination 
intensity level mapping function preventing the generation of any saturated pixels and 
providing a dynamic range of image data enabling each level in the histogram to be realized 

25 by the digital imager. The transfer control function generation circuit determines a second 
transfer control function based on the determined second illumination intensity level 
mapping function and imposes the determined second transfer control function upon a pixel 
of the digital imager. 

A twelfth aspect of the present invention is for determining transition points between 

30 a plurality of discrete transfer control functions forming a composite transfer control 
function. The system includes an exposure controller to determine an integration time; an 
illumination intensity level mapping controller to determine an illumination intensity level 
mapping function; and a transfer control function generation circuit to determine a 
composite transfer control fimction based on the determined integration time and 

35 determined illumination intensity level mapping function and to determine a timing of each 
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transition point between a plurality of discrete transfer control functions frona the 
determined integration time and the determined illumination intensity level mapping 
function. 

5 BRIEF DESCRIPTION OF THE DRAWINGS 

The present invention may take form in various components and arrangements of 
components, and in various steps and arrangements of steps. The drawings are only for 
purposes of illustrating a preferred embodiment and are not to be construed as limiting the 
present invention, wherein: 
10 Figure 1 shows a linear transfer control function with an integration time near 

maximum integration time; 

Figure 2 shows a linear transfer control function with a decreased integration time; 

Figure 3 illustrates mapping of pixel output voltage versus illumination intensity of 
the imaged scene using different integration times; 
15 Figure 4 illustrates a stepped or piecewise discrete-time transfer control function; 

Figure 5 illustrates mapping of pixel output voltage versus illuminadon intensity of 
the imaged scene using a linear transfer control function and a stepped or piecewise 
discrete-time transfer control function; 

Figure 6 is a histogram of illumination intensity levels of an actual scene to be 
20 imaged; 

Figure 7 is a histogram of illumination intensity levels of the imaged scene of Figure 
6 without conventional corrections; 

Figure 8 is a histogram of illumination intensity levels of the imaged scene of Figure 
6 using conventional corrections; 
25 Figure 9 is a histogram, enabled by the present invention, of illumination intensity 

levels of the imaged scene of Figure 6; 

Figure 10 is a block diagram of the adaptive sensitivity, on a pixel-by-pixel basis, 
control system according to the concepts of the present invention; 

Figure 1 1 is a flowchart showing one implementation of adaptive sensitivity control, 
30 on a pixel-by-pixel basis, according to the concepts of the present invention; 

Figure 12 is a flowchart showing another implementation of adaptive sensitivity 
control, on a pixel-by-pixel basis, according to the concepts of the present invention; 

Figure 13 shows calculating the timing of the transition points for a transfer control 
function according to the concepts of the present invention; 
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Figure 14 shows a desired map of output voltage versus illumination intensity 

level for a scene imaged to produce the transfer control function of Figure 13; 

Figure 15 illustrates a flowchart showing the operations of an illumination intensity 

mapping controller according to another embodiment of the present invention; and 
5 Figure 16 illustrates a flowchart showing the operations of a transfer control 

function generator circuit according to another embodiment of the present invention. 

DETAILED DESCRIPTION OF THE PRESENT INVENTION 

It is desirable to increase the dynamic range of the digital imager such that the low 
10 illumination intensity levels of the scene are emphasized, but also the high illimiination 
intensity levels of the scene are represented and no pixels become saturated during a given 
integration period. Moreover, it is desirable to utilize the entire dynamic range of the digital 
imager when imaging a scene, notwithstanding the actual range of illumination intensities of 
the scene. 

15 The present invention provides techniques and systems for adjusting the sensitivity, 

on a pixel-by-pixel basis, of a digital imager by adjusting the transfer control function 
(resulting in the production of the control signal) through determining an illumination 
intensity level mapping function and an integration time characteristic of the digital imager 
in a manner that enables full use of the entire output intensity level range of the digital 

20 imager without saturation of the digital imager. These techniques can be applied to a wide 
range of digital still and video camera configurations. 

In describing the present invention, various terms have been used. In this 
specification, a frame period is the time period needed to read all the pixels of a digital 
imager to produce one frame of an imaged scene. Integration time refers to the time period, 

25 within an overall fi-ame period, that a digital imager is receiving incident light or 
illumination firom a scene being imaged and producing a collectable charge or voltage in 
response to incident light or illumination of the scene being imaged. Illumination intensity 
level mapping function refers to the analog to digital conversion relationship between an 
analog signal rt;ceived from a pixel of the digital imager and a digital signal representing the 

30 illumination intensity received by the pixel. Output voltage versus illumination intensity 
level map is a graphical representation of the illumination intensity level mapping function. 
Transfer control function refers to the controlling of the charge dissipation from the pixel of 
the digital imager during the integration time period through timing manipulation and 
control signal voltage level or value manipulation. Transfer control ftmction generation 

35 refers to either the actual generation of the control signal that controls the charge dissipation 
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from the pixel of the digital imager or a signal that is used to generate the control 
signal that controls the charge dissipation from the pixel of the digital imager. 

More specifically, the present invention adjusts the integration time and manipulates 
the transfer control function (resulting in the production of the control signal) to realize a 
5 histogram of the imaged scene, as illustrated in Figure 9. In Figure 9, using a pixel-by-pixel 
sensitivity adjustment, the entire histogram is made to fit within the range of the sensor such 
that no saturation results while keeping the image average at 50 to ensure the image is 
bright enough to be visually appealing and thus not reducing the signal to noise ratio in the 
pixels of lower illumination intensities where the hmnan eye is more sensitive to noise. 

10 With the techniques of the present invention, the low illumination intensity levels of 

the scene are emphasized, and the high illumination intensity levels of the scene are 
discemable, with no pixels becoming saturated diuing a given frame period, F, 

The present invention varies the sensitivity of each pixel of a digital imager so that if 
a scene has a wide dynamic range, the digital imager can produce a non-linear mapping of 

15 the output vohages from the digital imager to digital illumination intensity levels or values 
representing the imaged scene. According to the concepts of the present invention, the 
output voltages representing the higher illumination intensity levels of the scene, where the 
eye is less sensitive to quantization noise, are non-Unearly mapped to illumination intensity 
levels such that the resolution of the higher illumination intensity levels are compressed. 

20 Moreover, the output voltages representing the low illumination intensity levels of the scene 
are amplified to ensure the image is bright enough. The results are imaged scenes with 
appropriate brightness and no saturation. 

The present invention dynamically adjusts the mapping of the output voltages to 
imaged scene illumination intensity levels, thereby continually compressing or 

25 decompressing the resolution of the high illumination intensity levels of the scene, so as to 
change or unequally distribute the amount of quantization noise for different levels of light 
(illumination) intensity. Under a completely linear mapping, all possible illimiination 
intensities of the imaged scene would have the same resolution or amount of quantization 
noise. 

30 The present invention selectively decreases or compresses the resolution of higher 

illumination intensity levels, by non-linearly adjusting the mapping of the output voltage 
versus illumination intensity level of the imaged scene map, to increase the saturation point. 
The result is that given a scene with a wide dynamic range, the output voltages associated 
with the low illumination intensity level pixels can be amplified enough to have the desired 

35 resolution and the output voltages associated with the high illiunination intensity level 
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pixels can be mapped to compress the resolution of the high illumination intensity 
levels to fit within the range of the analog to digital converter associated with the digital 
imager and avoid saturation. To realize this, in accordance tlie present invention, a 
desirable illumination intensity level mapping is automatically determined and the 
5 determined illumination intensity level mapping is converted into a transfer control function 
(resulting in the production of the control signal) for the digital imager. The transfer control 
function (resulting in the production of the control signal) controls the production or 
accumulation of charge in the photosensitive element, thus controlling the output voltage of 
the pixel. 

10 Figure 10 illustrates one embodiment of the present invention that emphasizes the 

low illumination intensity levels of the scene, and discerns the high illumination intensity 
levels of the scene, with no pixels becoming saturated during a given Jframe period, F. As 
shown in Figure 10, a system for adaptively controlling sensitivity, on a pixel-by-pixel 
basis, of a digital imager includes a digital imager 1, an illumination intensity level mapping 

1 5 controller 3, an exposure controller 5, and a transfer control function generation circuit 7. 

The illumination intensity level mapping controller 3 determines a number of pixels 
of imaged scene having illumination intensity levels within a first defined range of 
illumination intensity levels. The illumination intensity level mapping controller 3 also 
determines an illumination intensity level mapping function based upon the determined 

20 number of pixels within the first defined range of illumination intensity levels. 

The exposure controller 5 determines a number of pixels of the imaged scene having 
illumination intensity levels within a second defined range of illumination intensity levels 
and determines an integration time, T\nu based upon the determined number of pixels having 
illumination intensity levels within a second defined range of illumination intensity levels. 

25 The transfer control function generation circuit 7 determines a transfer control 

function based on the determined illumination intensity level mapping function and 
integration time. Tint. The transfer control function generation circuit 7 further imposes the 
determined transfer control function upon a pixel of the digital imager. 

An example of a transfer control function generation circuit is described in co- 

30 pending US Patent Application, Patent Serial Number 09/916,822, filed on July 27, 2001, 
entitled •TRECISE MOS IMAGER TRANSFER FUNCTION CONTROL FOR 
EXPANDED DYNAMIC RANGE IMAGING." The entire contents of co-pending US 
Patent Apphcation, Patent Serial Number 09/916,822, are hereby incorporated by reference. 
In a preferred embodiment, the digital imager 1 receives the transfer control 

35 function, and produces a control signal 10 therein in accordance with the received transfer 
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control function. 

It is noted that the transfer control function generation circuit 7 could also produce 
the control signal 10 directly and feed it to the digital imager 1, wherein the control signal 
10 is produced in accordance with the determined transfer control function based on the 
5 determined integration time. Tint, and detemiined illumination intensity level mapping 
function. 

The first defined range of illumination intensity levels is a range of illximination 
intensity levels including an illmnination intensity level representing pixel saturation, and 
the second defined range of illumination intensity levels is a range of illumination intensity 

10 levels including an illumination intensity level representing a minimum illumination 
intensity level that may be adjusted for pixel noise or pixel offset. 

The illumination intensity level mapping controller 3 determines a number of pixels 
of image data having illumination intensity levels within a first defined range of 
illumination intensity levels firom a fi-ame of pixels of image data created by the digital 

15 imager, firom a partial firame of pixels of image data created by the digital imager, from a 
defined area within a frame of pixels of image data created by the digital imager, or from a 
user-defined area within a frame of pixels of image data created by the digital imager. 

The illumination intensity level mapping function can be either calculated or 
selected from a plurality of pre-specified illumination intensity level mapping functions, 

20 preferably from eight pre-specified illumination intensity level mapping functions, based 
upon the determined number of pixels within the first defined range of illumination intensity 
levels. Moreover, the transfer control function can be either calculated or selected from a 
plurality of pre-specified transfer control fimctions based upon the determined illimiination 
intensity level mapping function and determined integration time. Tint- 

25 Furthennore, the determination of the transfer control function can realized during 

the generation of a second frame of image data that is immediate subsequent to the fi^ne of 
image data being used to make the determination. In this situation, the new transfer control 
function is imposed upon pixels during the generation of a third frame of image data, the 
third frame being immediate subsequent to the second firame of image data. 

30 On the other hand, if the determination is being made on only a partial frame, the 

determination of the transfer control function can realized during the generation of a first 
frame of image data which is the frame of image data having the partial frame of image. In 
this situation, the new transfer control fimction is imposed upon pixels during the generation 
of a second frame of image data, the second frame being immediate subsequent to the first 

35 frame of image data. 
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It is further noted that the frame period, F, or other frame parameters can be 
adjusted on the fly to enable implementation of the present invention. 

The illumination intensity level mapping function may be determined independent 
of, dependent upon, after, before, or substantially simultaneous with the determination of 
5 the integration time, Ti„,. Moreover, the illumination intensity level mapping fimction may 
be a composite function comprising individual discrete functions wherein each individual 
discrete illumination intensity level mapping function is determined based upon the 
determination of a number of pixels within a defined range of illumination intensity levels. 
Each individual illumination intensity level mapping function is associated with a defined 
10 range of illumination intensity levels. 

The illumination intensity level mapping controller 3, in one embodiment, 
determines a plurality of number of pixels, each determined number of pixels corresponding 
to one defined range of illumination intensity levels. In this embodiment, the illumination 
intensity level mapping controller 3 determines a plurality of illumination intensity level 
15 mapping functions, each determined illumination intensity level moping function 
corresponding to one defined range of illumination intensity levels. Each illumination 
intensity level mjq)ping function is determined based upon the determined number of pixels 
within the associated defined range of illimiination intensity levels. 

hi this embodiment, the transfer control function generation circuit 7 determines a 
20 transfer control function based on the plurality of determuied illumination intensity level 
mapping functions and imposes the determined transfer control function upon a pixel of the 
digital imager. 

hi another implementation of the system illustrated in Figure 10, the illumination 
intensity level mapping controller 3 determines a number of saturated pixels and selects a 
25 fiirst illumination intensity level mapping function when the determined number of saturated 
pixels is above a first threshold. The illumination intensity level mapping controller 3 
determines a number of pixels having iUumination intensity levels within a defined range of 
levels and selects a second illumination intensity level mapping function when the 
determined number of pixels is below a second threshold. The transfer control function 
30 generation circuit 7 determines a transfer control function based on the selected illummation 
intensity level mapping function and imposes the determined transfer control fimction upon 
a pixel of the digital imager. 

The transfer control fimction may comprise a plurality of discrete transfer control 
fiinctions, preferably eight discrete transfer control fimctions. hi this embodiment, the 
35 determined illumination intensity level mapping function comprises a pluraUty of discrete 

15 
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illumination intensity level mapping functions, preferably eight discrete 
illumination intensity level mapping functions. Each discrete transfer control function is 
determined based on one of the plurality of distinct illumination intensity level mapping 
functions. 

5 Each discrete illumination intensity level mapping function may be a linear 

illumination intensity level mapping function. The plurality of discrete linear illxmiination 
intensity level mapping functions form a composite piece-wise linear illumination intensity 
level mapping function, the composite piece-wise linear compression being the determined 
illumination intensity level mapping function. The detemiined composite illumination 
10 intensity level mapping function is a nearly logarithmic illumination intensity level mapping 
function. 

In a further implementation, the illumination intensity level mapping controller 3 
selects a first illumination intensity level mapping function, and the transfer control function 
generation circuit 7 determines a first transfer control function based on the selected first 

15 illumination intensity level mapping function and imposes the determined first transfer 
control function upon a pixel of the digital imager. The illumination intensity level 
mapping controller 3 then detemiines a histogram of illumination intensity levels of pixels 
of the imaged scene being generated by the digital imager having the determined first 
transfer control function imposed thereon and determines an illumination intensity level 

20 maximum, the illumination intensity level maximum representing a greatest illumination 
intensity level for a pixel in a sample forming the histogram. The illumination intensity 
level mapping controller 3 determines a second illumination intensity level mapping 
function, based on the detemiined intensity level maximum, the second illumination 
intensity level mapping function preventing the generation of any saturated pixels and 

25 providing a dynamic range of image data enabling each level in the histogram to be realized 
by the digital imager. The transfer control function generation circuit 7 then determines a 
second transfer control function based on the determined second illumination intensity level 
mapping function and imposes the determined second transfer control function upon a pixel 
of the digital imager. The first illxmiination intensity level mapping function represents a 

30 maximum compression of the resolution of the high illumination intensity levels. 

As noted above, one way of manipulating the transfer control function is to use a 
stepped or piecewise discrete-time transfer control function. By using a stepped or 
piecewise discrete transfer control function, the map of the pixel output voltage versus 
illumination intensity can be modified to enable a wider range of possible illumination 

35 intensity levels before saturation, while emphasizing low illumination intensity levels. 



wo 02/056584 PCTAJS02/00551 

However, to use a stepped or piecewise discrete-time transfer control 
function, each stepped transfer control function or control signal level must be determined 
as well as the timing of the transition points between discrete transfer control functions or 
discrete control signal levels. 

In the preferred embodiment of the present invention, the transfer control function is 
a stepped or piecewise discrete-time transfer control function, thereby producing a control 
signal that is a stepped or piecewise discrete-time control signal 

When the transfer control function generation circuit 7 receives Ti„t, tiie first 
transition point firom a first discrete transfer control function to a discrete second transfer 
control function of the composite transfer control function is immediately set. The 
illumination intensity level mapping function, g, determines the actual discrete transfer 
control functions, wherein the actual discrete transfer control functions determine each of 
the actual discrete control signal levels. 

In the example discussed below, the illumination intensity level mapping function, 
g, is derived from the desired map of output voltage versus illumination intensity level of 
the imaged scene illustrated m Figure 14. As seen in Figure 14, the desired map of output 
voltage versus illummation intensity level of the imaged scene is a piece-wise discrete 
mapping, with each discrete mapping corresponding to a different compression of the 
resolution of the associated illumination intensity level range. As noted below, each 
discrete compression of the resolution of the associated illumination intensity level range 
corresponds to a level, bi, in the transfer control function. As the compression of the 
resolution of the associated illumination intensity level range is increased, the 
corresponding level, bi, is decreased. 

As noted above, in a preferred embodiment of the present invention, the transfer 
control function is a discrete piece-wise linear function, having eight pieces or steps formed 
by seven transition points, that produces a nearly logarithmic response. 

An example of a stepped or piecewise discrete-time transfer control function, 
according to a preferred embodiment of the present invention, is illustrated in Figure 13. In 
Figure 13, the timing of the transition points, t, is set forth on the horizontal axis, and a 
level, b, corresponding to the transfer control function, which corresponds to an amount of 
the subsequently produced control signal being stepped down from a reset level, is set forth 
on the vertical axis. 

The stepped or piecewise discrete-time transfer control function in one embodiment 
of the present invention, as illustrated in Figure 13, is realized by: 
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1. Setting the slope of region from the first transition point, ti, to the second 
transition point, t2, to be m; 

2. Setting the slope of the region from the second transition point, ta, to the third 
transition point, ts, to be gm; 

5 3. Setting the slope of the region from the third transition point, ta, to the fourth 

transition point, U, to be g^m; 

4. Setting the slope of the region from the fourth transition point, to the fifth 
transition point, ts, to be g^m; 

5. Setting the slope of the region from the fifth transition point, ts, to the sixth 
10 transition point, t6, to be g "^m; and 

6. Setting the slope of the region from the sixth transition point, t6, to the seventh 
transition point, t-j or Tmax, to be g ^m. 

If more than seven transition points are desired, the slope of the region between 
subsequent transition points would continue to increase by a factor g. 

15 With such an approach, when the exposure controller 5 specifies the desired 

integration time. Tint, and the illumination intensity level mapping controller 3 specifies the 
desired illumination intensity level mapping function, g, the stepped transfer control 
function is completely constrained; however, it is still necessary to determine when the 
transition points between the discrete transfer control functions occur during tlie overall 

20 frame period, Tmax. 

As noted above, in this preferred embodiment of the present invention as illustrated 
in Figure 13, there are seven transition points. These seven transition points can be 
detemiined by solving the resulting sets of simultaneous equations: 

to ~ Tmax ~ Tint 

25 m = (bi-bo)/To 

mg = (b2-bi)/T, 
mg^ =(b3-b2)/T2 
mg^=(b4-b3)/T3 
mg'*=(b5-b4)/T4 

30 mg^=(b6-b5)/T5 

mg^=(b7-b6)/T6 
T™« =to+To+Ti+T2+T3+T4+T5+T6 



35 



The above equations use the variables all referenced in Figure 13. In a preferred 

embodiment, it is assumed that the levels, bo through b?, corresponding to the transfer 
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control function, which individually correspond to an amount of the subsequently 
produced control signal being stepped down from a reset level, are equally spaced by a 
quantity Ab (except that (b?- b6= 2Ab), the above equations, solving for To, Ti, T2, T3, T4, 
andTs yield 

To = (g V( i + g'+ g'+ g + 2) * Tin, 

Ti=(To)/g 
T2=(T,)/g 

T3 = (T2)/g 
T4=(T3)/g 
T5 = (T4)/g 

With the solving for To, Ti, T2, T3, T4, and T5, the seven transition points; to, ti, t2, t3, 
t4, ts, and te; are: 

to ~ Tmax " Tint 

ti=to+To 

t2=ti+Ti 
t3=t2+T2 
t4=t3+T3 
t5=t4+T4 
t6=t5+T5 

It is noted that the levels, bi, need not be evenly spaced to find the transition points. 
The equal spacing of b is just an example of creating a transfer control fimction according to 
the concepts of the present invention. 

In a preferred embodiment of the present invention, the determination of the timing 
of the transition points is realized by dedicated hardware due to the repetitive nature of the 
calculation. Once To is found, one simply does a divide by the same denominator five more 
times. &i the preferred embodiment, To is not calculated by dedicated hardware due to 
inefficiencies, thus the number of possible g's is Innited and a table lookup is used to find 
To. 

In this embodimait, the transition points between the plurality of discrete transfer 
control fonctions, which form a composite transfer control function, are determined 
following the determination of an integration time. Tint, and an illumination intensity level 
mapping fianction, the illumination intensity level mapping, function providing the proper 
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compression of the resolution of the high illumination intensity levels of imaged scene 

and proper emphasis of the low illumination intensity levels of imaged scene. 

In a preferred embodiment, the composite transfer control function has eight discrete 
transfer control functions and seven transition points. The first transition point is equal to a 
difference between a maximum possible integration time, Tmax> or frame period, F, and the 
determined integration time. Tint. Subsequent transition points, assuming that the levels, bj, 
are equally spaced by a quantity Ab (except that (b? - be = 2Ab), are equal to a sum of all 
previous transition points and a time Ts = ((g""')/((g"'^+g"'^+. . .+gVg+2)(g^P^))*Tint, where g 
is equal to the determined illumination intensity level mapping function, n is equal to a total 
number of transition points, p is equal to a positional number of the discrete transfer control 
function being calculated, and Tint is equal to the determined integration time. If the levels, 
bi, are not equally spaced by a quantity Ab, the equation would be slightly different 
depending upon the spatial relationship between levels, bi. 

Figure 15 illustrates a process utilized by an illumination intensity mapping 
controller of a preferred embodiment of the present invention. As illustrated in Figure 15, 
the illumination intensity mapping controller takes the imaged scene from the sensor and 
counts the number of saturated pixels. If there are too many saturated pixels, the 
illumination intensity mapping controller increases the amount of compression upon the 
resolution of the liigh illumination intensity levels. The illumination intensity mapping 
controller also counts the number of pixels in the upper region (bin) of a histogram of the 
imaged scene to determine if the imaged scene fills the entire range of the histogram. If 
there are not enough pixels in the upper region (bin), the illumination intensity mapping 
controller decreases the amount of compression upon the resolution of the high illumination 
intensity levels. The illumination intensity mapping controller continues this process until 
there are no saturated pixels and the histogram is completely used. 

This approach works well for video applications in that the illumination intensity 
mapping controller can continuously run without having to drastically change the 
illimiination intensity mapping function to determine how to best set it. If the number of 
illumination intensity mapping steps is minimized xmder the constraint of stability, eight 
illumination intensity mapping steps are sufficient. This allows the illumination intensity 
mapping function to be set within 16 frames as a maximum. 

Figure 16 illustrates a process utilized by a transfer control function generator circuit 

of a preferred embodiment of the present invention. As illustrated in Figure 16, the transfer 

control function generator circuit sets the initial level, b, of the transfer control function to 

bo. A step value, i, is set to a value of 1. The timing of the transition point, ti, is calculated. 

20 
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The transfer control function generator circuit goes into a standby state, sending the 
last establish level, b, to the digital imager, until the calculated transition point, ti, is 
realized. Wlien the calculated transition point, tj, is realized, a new level, b, is set equal to 
bi+i. The step value, i, is incremented by 1. The calculating of the transition point, ti, and 
5 setting of the level, b, is repeated until the step value, i, is equal to 6. 

It is to be recognized that the present invention contemplates both employing 
preprogrammed illumination intensity level mapping functions as well as computing custom 
illumination intensity level mapping functions on the fly during a given frame analysis. The 
use of preprogrammed illumination intensity level mapping functions can be preferred for 

10 applications in which computational power is to be minimized. Real time computation of 
illumination intensity level mapping functions can be preferred for applications in which 
flexibility in sensitivity control is desired. 

For example, given a frame analysis in which a large number of pixels have a 
particular intensity level, a customized illumination intensity level mapping function 

15 computed to allocate a large number of output levels to that common intensity level would 
be optimal for efficiently parceling the output range levels. Similarly, computation of a 
customized illimiination intensity level mapping function can be optimal for allocating a 
large number of output levels to the intensity range of a particular physical region of pixels 
of an image frame. Such a region may be of interest for its inclusion of, e.g., moving 

20 objects, or in general, objects of interest or importance in a scene. The present invention 
further contemplates use of both preprogrammed and custom, on-the-fly illumination 
intensity level mapping functions for a single application, whereby the benefits of both 
approaches can be obtained. 

The adaptive sensitivity control, on a pixel-by-pixel basis, of the present invention 

25 preferably is implemented to operate automatically, without the necessity for user control or 
input, to enable automatic, real time, adaptive adjustment of the digital imager for 
maximizing image quality. 

Figure 11 illustrates another embodiment of the present invention. As illustrated in 
Figure 11, in a first step, the integration time. Tint, of the digital imager is controlled. 

30 Integration time control is specifically implemented by controlling the number of pixels 
associated with low illumination intensity levels. This has the effect of shifting the dynamic 
range of the digital imager relative to the scene illmnination intensity range. 

In one example, intensity levels 0 through 3 are defined as a specified low intensity 
level range. Given that the low intensity level range includes levels 0 through 3, this 

35 accounts for 0.1% of the total number of intensity levels of a 4,096 output intensity levels 

21 
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imager. The integration time is controlled in accordance with the present invention so 

that the same percentage, 0.1%, of the total number of pixels, i.e., 0.1% of a 307,200 pixel 

imager, or 307 pixels, have the intensity levels of 0, 1, 2, or 3. This acts to enforce a 

condition whereby the entire 4096 levels are efficiently parceled across the intensity range 

5 of the scene. 

Preferably, the number of pixels selected to be associated with the low intensity 
level range is defined as a range; e.g., in the example just given, 307 +/- 30 pixels are 
preferably assigned to the range. The use of a number range of pixels enables the 
introduction of hysteresis, which stabilizes the system. Another way to introduce the 

10 hysteresis is to use an intensity level range. For example, if the number of pixels at the 
intensity levels of 0 or 1 is less than 307, in response, the exposure time can be decreased. 
Similarly, if the number of pixels at the intensity levels of 0, 1, 2, 3, or 4 is more than 307, 
then the exposure time can be increased. Li this example, the threshold value for the 
number of pixels is 307 and does not have any ranges, but the intensity level itself has a 

15 range from "0 or 1" to "0, 1, 2, 3, or 4". This introduction of intensity level ranges also 
provides hysteresis. The low intensity level range is preferable small, ' and typically the 
smaller the better, except that if too small, e.g., including only the 0 level, the algorithm 
may not be robust to noise levels or to the occurrence of faulty pixels. 

In the first step of Figure 1 1, the number of pixels of the frame that are found to be 

20 associated with the defined low intensity ranges are counted from the frame illimiination 
intensity histogram. If the number of low intensity range pixels is larger than that number 
corresponding to the percentage of low intensity ranges as a function of the total number of 
intensity levels, the integration time of the digital imager is increased. Conversely, if the 
number of pixels is smaller, the digital imager integration time is decreased. 

25 In step 2 of Figure 11, the illumination intensity level mapping fianction is adjusted. 

This is specifically realized by selecting an illumination intensity level mapping fimction to 
make a correspondence between a number of pixels and a high illumination intensity range. 
This correspondence is particularly made so that the distribution of pixels goes to zero at the 
saturation level of the digital imager, as shown in Figure 9. The higih illumination intensity 

30 range can be defined as, e.g., the intensity levels 4,095, 4,094, 4,093, and 4,092, given 4096 
output levels. For this example, the number of pixels to be associated with these output 
levels is then .1% of the total number of pixels. This pixel number can then be defined as 
the range between 277 and 337 pixels, for example, for a 307,200 pixel imager. 

With this step completed, the transfer control fimction of the digital imager is 

35 adjusted such that the digital imager output levels are adjusted in the maimer illustrated in 
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Figure 9. It is recognized in accordance with the present invention that the step of 
adjustment of imager integration time can be carried out after or in parallel with the step of 
adjustment of the level of digital imager control signal. The order of steps shown in the 
flowchart of Figure 11 is an example only. 
5 A simple example is described below to illustrate the concepts of the present 

invention. 

Consider an example digital imager having a pixel array size of 4x4, where for a 
given scene being imaged in a frame under analysis, all the pixels in each row have the 
same intensity level, and where the intensity level of each pixel is output in 4-bits ranging in 
1 0 output levels between 0 and 1 5 . 

Given further an example histogram of scene intensity levels to be imaged by the 
particular pixels of the digital imager is as follows: 

3 for the first pixel row, 
15 6 for the second pixel row 

300 for the third pixel row, and 
600 for the fourth pixel row. 

In other words, the scene intensity level ratios between the pixel rows are: 

20 

1 :2 for the first and second pixel rows, 
1 :50 for the second and third pixel rows, and 
1 :2 for the third and fourth pixel rows. 

25 In a conventional sensitivity control technique, in order to reduce the intensity level 

for the fourth row firom 600 to 15, which is the highest intensity level capable of the digital 
imager, the intensity of each pixel would need to be reduced by a factor of 40 by decreasing 
the integration time, with the new intensities after tiie integration time adjustment given as: 

30 0 (=1/40) for the first pixel row, 

0 (=2/40) for the second pixel row, 
8 (=300/40) for the third pixel row, and 
15 (=600/40) for the fourth pixel row. 
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Note that this results in a loss of the original intensity difference between the 

first and second pixel rows, and therefore results in a poor output image. 

This deficiency is overcome by the technique of the present invention. Following 

the technique of the present invention described above, the intensities of the pixels are 

S adjusted as follows. 

(a) Each illimiination intensity level is divided by a factor of 3, by reducing the 
integration time, in order to bring the lowest illumination intensity level of the scene being 
imaged to 1, the designated low illumination intensity level of the digital imager. The new 
illumination intensity output level of pixel rows one-four are then given as l(=3/3), 2(=6/3), 

10 100 (=300/3), and 200 (=600/3), respectively. 

(b) Each illumination intensity level is subjected to a non-linear illumination 
intensity level mapping function. The illumination intensity levels of 1 and 2 stay 
substantially the same, but the resolution of the highest illumination intensity level pixel 
region, having the intensity level 200, is compressed strongly to map highest illumination 

15 intensity level of the scene being imaged to the maximum intensity level 15 of the digital 
imager. The resolution of the high illumination intensity level pixel region, just below the 
highest illumination intensity level pixel region, is compressed not as strongly, and 
therefore the illumination intensity level of the third pixel row is set at 9- 

With these steps, the technique of the present invention converts the original pixel 

20 row illumination intensity level distribution of 1, 2, 300, and 600 Gx>m the scene into imager 
output levels of 1, 2, 9, 15. The original condition in which the four pixel rows have 
different illvunination intensity levels is preserved, while the image illumination range is 
fully mapped to the output range characteristic of the digital imager. 

The present invention contemplates a wide range of implementation alternatives for 

25 accommodating various operating considerations. For example, the computational load of 
the method described above can be relatively large due to the use of an illumination 
intensity level histogram calculation. A number of alternative methods can be employed to 
reduce the computational load, each at some cost of performance degradation. It is foimd, 
however, that the most prominent performance degradation is generally manifested only as 

30 a longer settling time when the illumination intensity of a scene is changing significantly; 
such is typically considered to be an acceptable condition. 

In a first example alternative technique, the number of illumination intensity levels 
of the histogram calculation is reduced. For example, 4,096 original illumination intensity 
levels can be replaced with a smaller number, e.g., four, illumination intensity level groups. 

35 The illumination intensity level groups 1, 2, 3, and 4 each can be mapped to, e.g.. 
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iUumination intensity levels of 0-523, 524- 1,047, 1048-1,571, and 1,572-4,095, 
respectively. This grouping of illumination intensity levels significantly reduces the needed 
computational load. 

In a further alternative technique, the histogram calculation can be eliminated. In 
5 this technique, the pixel illumination intensity distribution is specified to be even across 
4095 levels. In this case, 75 pixels are designated for each illumination intensity level for a 
640x480 pixel array, because 640x480/4,095=75. Therefore, 150 (=75x2) pixels are 
designated for the illumination intensity levels 0 and 1 of a low illumination intensity range. 

Figure 12, illustrates a further embodiment of the present invention for estabhshing 
10 the integration time. As shown in Figure 12, the total number of pixels for the illumination 
intensity levels 0 and 1 are counted, and noted as NumberDark (ND). After determining 
ND, the process determines the adjustment to the integration number to reduce the value of 
ND. The method for increasing the integration time to reduce ND is as follows: 

15 (1) If 1 65<ND<=1 80, increase the integration time by 1 0%; 

(2) If 180<ND<=195, increase the integration time by 20%; 

(3) If 195<]SID<=210, increase the integration time by 30%; 

(4) If 210<ND<=225, increase the integration time by 40%; 

(5) If 225<ND<=240, increase the integration time by 50%; 
20 (6) If 240<ND<=270, increase the integration time by 70%; 

(7) If 270<ND<=300, increase the integration time by 90%; 

(8) If 300<ND<=450, increase the integration time by 1 50%; 

(9) If 450<ND<=600, increase the integration time by 250%; 

(10) If 600<ND<=900, increase the integration time by 400%; 
25 (1 1) If 900<ND<=1 500, increase the integration time by 750%; 

(12) If 1500<ND, increase the integration time by 1,000%; and 
If ND is between 135 and 165, there is no change in integration time. 

If ND is less than 135, the following rules are instead applied to decrease iihe 
30 integration time: 

(1) If 120<ND<=135, decrease the integration time by 11%; 

(2) If 105<ND<=120, decrease the integration time by 25%; 

(3) If 90<ND<=105, decrease the integration time by 42%; 
35 (4) If 75<ND<=90, decrease the integration time by 66%; 

25 
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(5) If 60<ND<=75, decrease the integration time by 100%; 



(6) If 30<ND<=60, decrease the integration time by 233%; 

(7) If 25<ND<=30, decrease the integration time by 400%; 

(8) If 20<ND<=25, decrease the integration time by 500%; 



5 



(9) If 15<ND<=20, decrease the integration time by 650%; and 
(10) If ND<=15, decrease the integration time by 1,000%. 



Before, after, or simultaneous with this integration time control, the illmnination 
intensity level mapping ftmction of the digital imager is controlled so that the number of 

10 pixels that have the illumination intensity levels of either 4095 or 4094 is controlled to be 
between 135 and 165. The number of such 'iDright" pixels, high illvunination intensity level 
pixels, is counted and designated NumberBright (NB), and then the illumination intensity 
level mapping function is adjusted accordingly. In one example, the illumination intensity 
level mapping function is selected from a number, e.g., eight, different illumination 

15 intensity level mapping fimctions available. It is contemplated in accordance with the 
present invention that any reasonable number of functions can be provided. 

In this embodiment, the following rules are applied, based on the counted NB, to 
increase or decrease the compression of the resolution of the higji illumination intensity 
level through the illumination intensity level mapping function of the digital imager: 



20 



(1) If 165<NB<=200, invoke a stronger compression upon the resolution of 
the high illimiination intensity levels by one notch. 



(2) If 200<NB<=300, invoke a stronger compression upon the resolution of 
the high illumination intensity levels by two notches. 



25 



(3) If 300<NB<=400, invoke a stronger compression upon the resolution of 
the high illumination intensity levels by three notches. 



(4) If 400<NB<=500, invoke a stronger compression upon the resolution of 
the high illumination intensity levels by four notches. 



30 



(5) If 500<NB<=600, invoke a stronger compression upon the resolution of 
the high illumination intensity levels by five notches. 



(6) If 600<NB<=700, invoke a stronger compression upon the resolution of 
the high illumination intensity levels by six notches. 



(7) If 700<NB, invoke a stronger compression upon the resolution of the 
high illumination intensity levels by seven notches. 
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(8) If 120<NB<=135, invoke a weaker compression upon the 
resolution of the high illumination intensity levels by one notch. 

(9) If 100<]SIB<=120, invoke a weaker compression upon the resolution of 
the high illumination intensity levels by two notches. 

5 (10) If 80<NB<=1 00, invoke a weaker compression upon the resolution of the 

hig^h illumination intensity levels by three notches. 

(11) If 60<NB<==80, invoke a weaker compression upon the resolution of the 
high illumination intensity levels by four notches. 

(12) If 400<NB<=60, invoke a weaker compression upon the resolution of the 
10 high illiraiination intensity levels by five notches. 

(13) If 20<NB<=40, invoke a weaker compression upon the resolution of the 
high illumination intensity levels by six notches. 

(14) If 20NB, invoke a weaker compression upon the resolution of the high 
illumination intensity levels by seven notches. 

15 

These rules enable selection of an illumination intensity level mapping function that 
proportions the intensity levels of the pixels in the manner described earlier. 

In one embodiment of the present invention, a number of pixels of image data 
having illumination intensity levels within a fiirst defined range of illumination intensity 

20 levels is determined. From this determined number of pixels, an illumination intensity level 
mapping fimction is determined that is used to create a transfer control Amotion that can 
then be imposed upon a pixel of the digital imager. A number of pixels having illumination 
intensity levels within a second defined range of illumination intensity levels is determined 
to enabling the determination of an integration time. If the integration fimction is 

25 determined, the determination of the transfer control fimction is based on the determiaed 
illumination intensity level mapping fimction and the determined integration time. 

The determination of an illumination intensity level mapping function and 
integration time is repeated until a desired dynamic range is reaUzed. The first defined 
range of illumination intensity levels is a range of illumination intensity levels mcluding an 

30 illumination intensity level representing pixel saturation, and the second defined range of 
illumination intensity levels is a range of illumination intensity levels including an 
illumination intensity level representing a minimum illiraiination intensity level, wherein the 
minimum illumination intensity level can be adjusted for a pixel offset, residual pixel 
charge. 

27 
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The detennination of a number of pixels of image data having illumination 

intensity levels within a first defined range of illumination intensity levels can be fi-om a 

firame of pixels of image data created by the digital imager, firom a partial frame of pixels of 

image data created by the digital imager, firom a defined area within a fi:ame of pixels of 

5 image data created by the digital imager, or from a user-defmed area within a frame of 

pixels of image data created by the digital imager. 

The illumination intensity level mapping fiinction can be either calculated or 

selected from a plurality of pre-specified illxmiination intensity level mapping fimctions, 

preferably from eight pre-specified illumination intensity level mapping fimctions, based 

10 upon the detemiined number of pixels within the first defined range of illumination intensity 
levels. Moreover, the transfer control fimction can be either calculated or selected from a 
plurality of pre-specified transfer control fimctions based upon the determined illumination 
intensity level mapping function and determined integration time. 

Furthermore, the determination of the transfer control fimction can realized during 

15 the generation of a second frame of image data that is immediate subsequent to the firame of 
image data being used to make the detennination. In this situation, the new transfer control 
fimction is imposed upon pixels during the generation of a third frame of image data, the 
third frame being immediate subsequent to the second frame of image data. 

On the other hand, if the determination is being made on only a partial frame, the 

20 determination of the transfer control fiinction can realized during the generation of a first 
frame of image data which is the frame of image data having the partial frame of image. In 
this situation, the new transfer control fimction is imposed upon pixels during the generation 
of a second frame of image data, the second frame being immediate subsequent to the first 
frame of image data. 

25 The illumination intensity level mapping fimction may be determined independent 

of, dependent upon, after, before, or substantially simultaneous with the determination of 
the integration time. Moreover, the illumination intensity level mapping fimction may be a 
composite fimction comprising individual discrete fimctions wherein each individual 
discrete illumination intensity level mapping fimction is determined based upon the 

30 detemiination of a number of pixels within a defined range of illumination intensity levels. 
Each individual illumination intensity level mapping fimction is associated with a defined 
range of illumination intensity levels. 

In another embodiment of the present invention, a number of saturated pixels is 
determined and a first illumination intensity level mapping fimction is selected when the 

35 detemiined number of saturated pixels is above a first threshold. This embodiment also 
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determines a number of pixels having illumination intensity levels within a defined 
range of levels and selects a second illumination intensity level mapping function when the 
determined number of pixels is below a second threshold. The transfer control function is 
determined based on the selected illumination intensity level mapping function and imposed 
5 upon a pixel of the digital imager. 

Iq a third embodiment, the present invention determines a number of pixels of image 
data having illumination intensity levels within a first defined range of illimaination 
intensity levels, the first defined range of illumination intensity levels including an 
illimiination intensity level corresponding to a pixel saturation value and determines an 

10 illumination intensity level mapping function based thereupon. An integration time is 
determined based upon a number of pixels having illumination intensity levels within a 
second defined range of illumination intensity levels, the second defined range of 
illumination intensity levels including an illumination intensity level corresponding to a 
minimvim illumination intensity level. A transfer control function is created based upon the 

15 determined illumination intensity level mapping function and the determined integration 
time and imposed upon a pixel of the digital imager. 

The transfer control function comprises a plurality of discrete transfer control 
functions, preferably eight discrete transfer control functions. 

In a further embodiment, the present invention selects a first illumination intensity 

20 level mapping function and determines a first transfer control function based on the selected 
first illumination intensity level mapping function. The determined transfer control function 
is imposed upon a pixel of the digital imager. Thereafter, a histogram of illumination 
intensity levels of pixels of image data being generated by the digital imager having the 
determined first transfer control function imposed thereon is determined and an illumination 

25 intensity level maximum is determined, the illumination intensity level maximum 
representing a greatest illumination intensity level for a pixel in a sample forming the 
histogram. A second illumination intensity level mapping function is determined based on 
the determined intensity level maximum. The second illumination intensity level mapping 
function prevents the generation of any saturated pixels and provides a dynamic range of 

30 image data enabling each level in the histogram to be realized by the digital imager. A 
second transfer control function is determined based on the second illumination intensity 
level mapping function and imposed upon a pixel of the digital imager. The first 
illumination intensity level mapping function is a maximum compression of the resolution 
of the high illumination intensity levels of the imaged scene. 
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The present invention contemplates additional techniques for adaptively 

controlling the sensitivity, on a pixel-by-pixel basis, of a digital camera. For example, the 

integration time control techniques described above can be replaced or combined with any 

suitable mechanical iris control and/or intensity signal gain control. Mechanical iris control 

and intensity signal gain control can be employed to produce results similar to the 

integration control, but these controls produce some side effects also. If the mechanical iris 

is increased, for example, the depth of focus will be correspondingly decreased. If the 

signal gain is increased, as another example, the noise level, as well as the signal level will 

be correspondingly increased. 

The description jprovided above and accompanying diagrams are provided to 
demonstrate the broad scope of the adaptive sensitivity control contemplated by the 
invention. No particular implementation of the techniques of the invention is required; 
rather, any implementation that enables compression rate and integration time control of a 
digital imager to enable imager sensitivity control is contemplated by the invention. 

While various examples and embodiments of the present invention have been shown 
and described, it will be appreciated by those skilled in the art that the spirit and scope of 
the present invention are not limited to the specific description and drawings herein, but 
extend to various modifications and changes all as set forth in the following claims. 
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1 1. A method for determining transition points between a plurality of discrete transfer 

2 control functions forming a composite transfer control function, comprising: 

3 (a) determining an integration time; 

4 (b) determining a compression fimction; 

5 (c) determining a composite transfer control function based on the determined 

6 integration time and determined compression function; and 

7 (d) deteraiining each transition point between a plurality of discrete transfer control 

8 functions from the determined integration time and the determined compression function. 

1 2. The method as claimed in claim 1, wherein the composite transfer control 

2 function has eight discrete transfer control functions and seven transition points. 

1 3. The method as claimed in claim 1, wherein a first transition point is equal to a 

2 difference between a maximum possible integration time and the determined integration 

3 time. 

1 4. The method as claimed in claim 1, wherein a first transition point is equal to a 

2 difference between a possible maximum integration time and the determined integration 

3 time and a subsequent transition point is equal to a sum of all previous barrier break points 

4 and a time Ts where 

5 Ts is equal to ((g"-')/((g"*'+g""^+. . .+g'+g+2)(g^>)))*TiNT, 

6 g is equal to the determined compression function, 

7 n is equal to a total niunber of transition points, 

8 p is equal to a positional number of the discrete transfer control function being 

9 calculated, and 

10 Tnsrr is equal to the determined integration time. 

1 5. The method as claimed in claim 4, wherein a first transition point is equal to a 

2 difference between a possible maximum integration time and the determined integration 

3 time and a subsequent transition point is equal to a sum of all previous barrier break points 

4 and a time Ts where 

5 Ts is equal to ((g"-')/((g"*'+g""^+. . .+g2+g+2)(gCP))))*TiNT, 

6 g is equal to the determined compression function, 

7 n is equal to a total number of transition points, 
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8 p is equal to a positional number of the discrete transfer control function being 

9 calculated, and 

10 Tint is equal to the detennined integration time. 

1 6. A system for determining transition points between a plurality of discrete transfer 

2 control functions forming a composite transfer control function, comprising: 

3 an exposure controller, operatively connected to the digital imager, to determine an 

4 integration time; 

5 an illumination intensity level mapping controller, operatively connected to the 

6 digital imager, to determine an illumination intensity level mapping function; and 

7 a transfer control function generation circuit, operatively connected to the digital 

8 imager, said exposure controller and said illumination intensity level mapping controller, to 

9 determine a composite transfer control function based on the determined integration time 

10 and detennined illimiination intensity level mapping function and to determine each 

11 transition point between a plurality of discrete transfer control functions from the 

12 determined integration time and the determined illumination intensity level mapping 

13 function, 

1 7. The system as claimed in claim 6, wherein the composite transfer control 

2 function has eight discrete transfer control functions and seven transition points. 

1 8. The system as claimed in claim 6, wherein a first transition point is equal to a 

2 difference between a maximum possible integration time and the determined integration 

3 time. 

1 9. The system as claimed in claim 6, wherein a &st transition point is equal to a 

2 difference between a possible maximum integration time and the detennined integration 

3 time and a subsequent transition point is equal to a sum of all previous barrier break points 

4 and a time Ts where 

5 Ts is equal to ((g"-')/((g"-^+g"-'+. . .+g'+g+2)(g^^))*TiNT, 

6 g is equal to the determined compression function, 

7 n is equal to a total number of transition points, 

8 p is equal to a positional number of the discrete transfer control function being 

9 calculated, and 

10 Tint is equal to the determined integration time. 



32 



wo 02/056584 PCT/USO 2/00551 

1 10. The system as claimed in claim 9, wherein a first transition point is equal to 

2 a difference between a possible maximum integration time and the determined integration 

3 time and a subsequent transition point is equal to a sum of all previous barrier break points 

4 and a time Ts where 

5 Ts is equal to ((g"-' W+g""^+. . .+g'+g+2)(g^^))*TiNT, 

6 g is equal to the determined compression function, 

7 n is equal to a total number of transition points, 

8 p is equal to a positional number of the discrete transfer control function being 

9 calculated, and 

10 Tint is equal to the determined integration time. 
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